Method and device to pick up, transport and put down a load

ABSTRACT

A crane control based on quoted location data and observed ambient data provided by at least one identifier, the goods to be conveyed by the crane can be controlled to the location site. The identifier takes pictures of at least two difference areas, which is implemented so that the identifier is pivotable, or fixed, whereby the certain image area directed to the identifier is reflected from a reflecting surface.

BACKGROUND OF THE INVENTION

The objective of the invention is a method for conveying a load betweenlocation sites, preferably sub-methods for gripping the load, placingthe load on a desired site and for controlling the crane based oninformation received from the identification means, as well as anequipment therefor.

A crane is used to lift and move rolls, containers or correspondingproducts from one place to another with at least one grip member, e.g. aC-hook, or with two grip members placed on the opposite sides of theload to be gripped, or with several grip members. Bridge cranes, hostingcranes, is knuckle boom cranes can be moved fairly precisely on rails,but several factors, such as wind, stretching of the crane cable,swings, bending of the crane construction caused by the weight of theload to be lifted, cause trouble in gripping the load and moving it to adesired site. Piling of the load causes trouble when the first goods areplaced at the bottom of the load, or when stacking the outer portions ofthe pile, when there is not corresponding goods on the other side of thegoods to be stacked but an edge, e.g. the edge of a ship hold or a floor(empty). This has been a problem due to the lack of a suitableobservation means for this purpose to survey the desired site to obtainan image of the correct angle. All machines, ships, trains, trucks andmovable goods can temporarily be positioned in relation to each othereither by measuring electrically or also physically by a meteringdevice. A known satellite positioning is the GSP-method (GlobalPositioning System), in which the positioning of the gripper or themachine part in relation to the positioning satellites is implementedwith an accuracy of 0.1-1 meter. The GSP-positioning functions so faronly outdoors. This is not always functional or sufficiently accurate.

When loads are being conveyed on crane cables, the load comes into aswinging movement, which makes the work difficult. The swings have beentaken into consideration by using e.g. a synchronizer, whereby eachproduced change of acceleration is followed by another equally bigchange but in the opposite direction after a certain time period. Todamp the swing, an optimal speed profile is calculated for the motion,which eliminates the swing at the end of the motion and minimizes thetime used for the motion. Previously known solutions have thus definedthe swing equation of the load based on calculated values. The swing ofthe load can be controlled by the information. The swing control is infact based on the calculated default formula. No real time control isarranged. When current systems use different counterforces to absorb theload swing, the target site might drift elsewhere than to a certainsite, and a repetition of the same stop event at the same known targetsite seems fairly theoretical.

The swing absorption of the load should also function with a gantryrobot lifting pillar or other structure preventing the free swinging ofthe load and the gripper. The robot lifting pillar is assumed to berigid with a small load. When the bulk to be transferred is increased,bending is conducted also on the pillar by the load carrying structures,but these do not follow the mathematical harmonic swing formula, becausethe load carrying structures of the load act as springs. The arrangingof the swing absorption by some mathematical formula would thus requireempirical tests, as the spring constants etc. of the structures vary ine.g. a bridge construction according to how close the trolley is fromthe end carriers of the bridge. The above presented situations can alsobe managed by the invention.

The problem can e.g. be that when the crane driver obtains informationabout the transit distance of the cable, this does not generally enablehim to drive the load sufficiently accurate to the desired site, as e.g.the 10 ton and 30 ton load carried by the crane causes a bending of adifferent size on the crane bridge and also on the stretching of thecable. Changes in the loading platform of the goods (ship's draft) canalso cause problems to the crane driver. One solution to the abovementioned problems has been to e.g. identify lines marked in the groundor alike by a recognition means provided in the crane, based on whichthe load is transferred and the location becomes known. This ischaracteristic for container travelling gantry cranes. The markingscause additional work and the maintenance can be difficult.

SUMMARY OF THE INVENTION

An improvement is achieved by the invention to the above mentionedmatters. The invention is substantially characterized in what ispresented in the following specification regarding a method forconveying a load between location sites, preferably regardingsub-methods for gripping the load, placing the load on a desired siteand for controlling the crane based on information received from therecognition means, as well as regarding an equipment therefor.

A technical solution is presented both for the gripping situation of thegoods and for the transferring of the goods in the gripper to apre-known mathematical environment. All previously known solutions haveaimed at increasing the loading effectiveness with fixed recognitionmeans, sensors, cameras, which has required the use of differentauxiliaries, as presented above. The invention avoids marking of lines.The view angle of the camera used can further be selected and freelyadjusted. Only the certain area of the picture can be viewed. Theadvantage of the invention is also that the crane driver can be giventhe necessary control data so that the crane driver can concentrate ondriving. The load swing is according to the invention controlled almostin real time. Calculation of different swing equations or the like isavoided. The invention aims in fact essentially at real timeobservation, i.a. the location of the load in relation to the targetarea or a possible obstacle is known. One feature of the invention isalso that it improves the possibilities of preventing transport damageof the goods.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The invention is below described with reference to the specification andthe figures, wherein

FIGS. 1A and 1B show a basic picture of the conveying system;

FIGS. 2A through 2D illustrate as a basic picture the transferring ofthe load from one site to another and related physical phenomena;

FIG. 3 presents a method of defining the image area;

FIG. 4 presents a method of establishing the transition of the imagearea;

FIG. 5A and 5B show the swing motion of the load in relation to thetrolley;

FIG. 6 shows a picture of the camera reviewing area;

FIG. 7 shows different positions of the camera in relation to the loadto be transferred;

FIG. 8 presents one form of embodiment of the gripper and the cameras;

FIG. 9 shows an enlargement of the gripper according to FIG. 8;

FIG. 10 presents the turning equipment of the reflecting surface;

FIG. 11 presents a basic picture of the data management system;

FIGS. 12A through 12E presents a flow diagram of the swing damping fromits establishment to the crane or robot control;

FIGS. 13A and 13B are orthogonal views showing the measuring of thedistance between the gripper and the object by laser beam.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1A and 1B present a common lift arrangement between two or severalstacks of goods in a harbor container crane operation area from twodirections. In an automatic or semi-automatic transferring operation thelifting of the goods is implemented based on the typesetting figuresrecorded in the computer or logic memory. The typesetting figurecomprises the summing data of the reference points and the deviationfiles, which form the point space of the crane, defining the theoretical(pre-calculated) location place of the product in the stack 15, goodsvan 16 and ship 17. The theoretical location site is specified based onthe real picture given by the camera from where the information istransmitted to the crane control system in order to find the rightlocation site of the product. The points A, B, C, D and E depict thereference points of each pile to which the deviation is summed toprovide information about the individual position of each product inrelation to the reference point of the stack. In FIG. 1 E n,n orm,n,m=integer contains the theoretical data of the desired locationplace of each desired product, such as the container, reel, in relationto the reference point. Based on this data and the data provided by thecamera the grippers are controlled to place the product in the locationplace defined by data obtained in advance. Due to stretching of thecable, wind, etc. the location place has to be changed based on imagedata obtained from the camera. The crane is movable on a rail and/or bydifferent carrier, jib, etc. solutions. The crane can be provided withseveral shafts operating simultaneously. The crane can be manuallyoperated, semi-automatic or automatic.

For example, although the selected point would have been changed, e.g.the previous truck is loaded and a new truck has been driven close tothe truck loading place, to the place ‘1999’-point, the gripper startsloading the truck from the beginning, from the first record of thedeviation file. The vertical intersection point of the truck frontloading platform has been programmed into this deviation point as theentry system check point. Even if the position of the truck should beslightly different from the previous one, the gripper searches theloading platform front angle based on the camera picture data, and takesthe first container on the platform. The next container, in thedeviation point 2, focuses the camera on the vertical intersection pointof the first container observing simultaneously the height change thatoccurred in the height of the pattern recognition (about one containerupwards). Besides the management and control of the camera system basedon data saved in the memory, the position and deceleration points of thenew product to be transferred are compared to the previous product orproducts, which determine when the crane is to be operated slower, i.e.when approaching the location site of the load.

The gripper can be operated normally during the transfer when there isno sharp line, curve, etc. in the field of view but the picture isblurred. When a sharp uniform interface appears in the picture, thegripper motion is automatically stopped. When the deviation record hasbeen given a fixed, active area resulting from the physical position ofthe machine vision system, the malfunctions caused by the product in thegripper or other construction features of the crane are eliminated.

The bridge crane presented in FIG. 2A comprises a bridge 3 and ahorizontally moving trolley 4 thereon. The trolley 4 is moved in thedirection of the horizontal movement of the trolley, which causesswinging. The driving gear of the trolley comprises an electric motor,an electric current controlled brake and a suitable transmission gear,which are not shown. The trolley comprises a hoisting gear controllingthe cable motion. If reviewing a normal hand-operated container crane'stransferring of one container from pile 15 to the ship's hold 17, thedriver has to carry out several correction and pre-maneuver commands.Between the start command of the trolley and the actual starting of themovement of the trolley is a time delay of 20-500 ms depending on theconstruction and quality of the control system. When the driver changesthe control command, the trolley always follows after a time delay.After the attachment the hoisting and the horizontal transfer towardsthe target point starts. Because the gripper 21, 22 during the transferdoes not only swing according to the mathematical pendulum but also dueto wind power, bending and/or stretching of the crane constructions,etc. additional forces, the camera provides a new opportunity especiallywhen approaching the target area with the gripper 21, 22 and theproduct. The machine vision system due to the deviations and stockpoints has more known information about the target area compared to acompletely unknown target surrounding.

When the crane trolley arrives at the programmed target point theobjective is to create a situation where the swing of the grippers 21,22 would have been damped before the trolley stops. The target area isthen approached first utilizing the camera to damp the swing of theload, and especially in the last step to position the load in the actualtarget area, when from a certain front angle of the already positionedload, e.g. a container, the location site of the load to be transferredis recognized by a turning camera 1, 2. FIGS. 2B through 2D containdiagrams, of which the upper one, FIG. 2B, illustrates the driving speedas a function of the conveyed distance. The middle diagram, FIG. 2C,illustrates the swinging of the gripper 21, 22 without damping of theswing. The optimal driving instruction of the automation system shouldbe anticipated when driving the gripper 21, 22 to the target point, aspresented in the lowest diagram, FIG. 2D. The changes of the gripper's21, 22 (spreader) swing angle are presented in FIG. 2 also as loadmotions. The piles between which the transfer is made are marked byletters A, B and C. Different reference times relating to the motionstep are presented by t₁-t₉.

The damp of the swing of the load when the distance, i.e. heightinformation h=h₁−h₂, between the camera and the target is not known, canbe performed during the horizontal motion at the interval t₁ . . . t₇ byturning the camera 1, 2 of the gripper 21, 22 upwards and by reviewingthe crane main supports (or main support), the bridge or bottom surfaceof the trolley by an illuminated laser light recognition technique orthe like technique by placing the known shape above the gripper 21, 22in the known place. The actual transfer of the product to the targetarea would take place at the interval t. . . t₉. Although the abovementioned times overlap, the starting point of the approaching can bechosen freely if the system comprises several cameras. Four cameras canbe connected to the same control computer in present camera systems. Oneof these could then be directed upwards and two other would be in thegripper 21, 22 jaws. Another alternative e.g., however not equally good,is that the cameras are placed in the grippers, and one camera isadditionally provided in the trolley to observe the gripper and the loadswing. When the floor level or the upper surface level of the goods onthe floor and the features of the gripper are known, the accurate motionof the gripper in relation to the floor can be determined. Thisapplication has the numeric positioned height of the gripper from thecrane.

It is not necessarily essential that the damp of the load swing wouldactually be implemented from the area below the gripper 21, 22, but thatthe load swing damp has been considered in general.

Below is presented alternatives for determining the height of the cranegripper 21, 22 from the desired platform, i.e. the distance between thetarget area and the gripper. In the absence of the crane gripper 21, 22height information, the selected target area to be reviewed can bedetermined from a video picture using a combination of thecamera/cameras 1, 2 and the laser beam 32 according to FIGS. 13A and13B. The relative transition of the selected target area is calculatedand converted to the crane control data based on two successive or veryclose video picture samples of the machine vision camera (RGB,CCD-camera) image area. The laser beam 32 from the semiconductor lasersource provided in the gripper, is set in the known angle α in relationto the center axis of the cameras. As the laser beam does not dispersein the same way as a normal light beam, the shape of the light reflectedfrom the laser beam reflection point to the camera 1, 2 is constant andeasy to retrieve reliably even from a big picture material. The distanceof the gripper 21, 22 from the target, measure h, can be calculated inFIGS. 13A and 13B, when the laser light reflection is found by thecomputer 5 from the picture material provided by the camera. Dependingon the deviation s from the selected point of the camera image area, thedistance of the light point from the gripper can be exactly calculated,as the angle a is known. The camera in use provided with a zoomobjective is calibrated when the system is taken into use, whereafterthe height of the gripper from the object can always be calculated whenthe magnification ratio of the zoom objective has been considered.

The utilization of the dimensions of a known product or the automaticaccuracy functions of some machine vision systems, when the focusingdistance of the objective is known, e.g. Cognex Auto-Focus-function,enables also the damp of the load swing without a numeric positioning inthe hoisting device, or a laser light source according to the previousexample. The height data h can consequently be calculated by theprogram. When from the first full resolution video image area is “cut” ae.g. 64×64 pixel image area, this can be positioned from the next or thefollowing pictures at a time delay of approx. 30-50 ms per picture to bereviewed (FIG. 3, 4). This results in the situation where the transitionof the cut image area in each chosen new video picture can be determinedcompared to the original or a comparable picture. When the position ofthe pixels in relation to each other is known, the direction and speedof the swing of the crane gripper in relation to the ground, the trolleyand/or the bridge can be determined. The measuring can start any timeand the information about the time slot between the shots of thepictures to be compared is thus based on practical tests. If e.g. someproduct or the real size of its feature in the image area is known,which is the case when transferring products of standard type and sizeto or from the store, no height measuring is required, because thepixels of the picture can be changed into a relative motion when thedistance of the gripper in relation to the target area has beendetermined by means of the known product or feature. If a combination ofa camera and a laser light source is used, a sample of the laser lightreflection ambience is taken using the above described video picturecutting method, and the transition of this cut area is compared withsuccessive and very close picture shots.

The information obtained from the camera picture is illustrated in FIGS.3 and 4. In FIG. 3 the picture is taken from the area 100 below thecamera with the camera 1 or 2 provided in the gripper 21, 22, thusshowing part of the area below the camera. From the picture shown by thecamera is determined the searched digital image area 102, which is partof 101, and which is saved in the computer digitizing card 10 memory.The searched area should be so located that the crane or the robot doesnot manage to drive out of the screen or that the picture angle to thearea does not change too much, so that the lighting shadows could changeessentially. The deviations e.g. X′ and Y′ of the searched image areaare calculated from some image area point of the camera, e.g. from thevertical intersection point 104.

After a while, e.g. after 10-500 ms, a new picture 101 is taken with thesame camera 1 or 2 as the previous picture. The digital image area 102saved in the memory based on the crane or robot motion has moved inrelation to the camera 1 or 2. The searched field 102 can be searched bythe computer digitalizing card from the whole display or only from partof the camera display. When the machine vision system has searched allpossible digital image area data from the defined searched area, thesystem informs the compatibility quality for each new target found ofthe picture field 101 at the digital image area 102 of the originalpicture. As the searched field 102 has to be individual and sufficientlylarge, there are not in practice two or more alternatives. If the systeminforms several fields to be suitable as the searched area, thequalitatively most suitable area is chosen. When found, the location ofthe area in the camera image area 101 can be calculated, thus providingX″ and Y″. If the relative transition of X″ and Y″ compared to the X′and Y′-measurements of the previous picture exceeds the defined limitand there are several such found fields, the qualitatively next one ischosen, etc. Finally, if the results are not reasonable, the positioningof the gripper swing is started all over (FIG. 3).

As the distance to the object is known either by the crane or robotdigital positioning system or the known laser beam 32 mounted ininclined angle in the immediate vicinity of the camera, the motiondirection, speed and acceleration of the cameras 1, 2 and thus that ofthe gripper can be determined by the relative motion distancedifferences of X′ and X″ and Y′ and Y″ calculating from the pixels.

The searched digital image area data 102 must not contain lightreflection of the laser light source 32 attached to the gripper 21, 22(or it should come outside the field), because this high intensity lightmoves in relation to the searched image area 102 along with the cameras1 and 2 and impairs the success of the search. If the crane is providedwith a numeric positioning system, the motion speed of the gripper 21,22 in relation to the crane or the robot can be determined and a realtime correction of the gripper swing can be made by adjusting the speedof the crane or the robot. When the crane or the robot is operated athigher speed, the relative motion of the gripper is damped in relationto the motion speeds of the supporting structures of the crane or therobot, e.g. the bridge during measurement, and at low operating speed inrelation to the ground and the actual target area. Especially, if thecrane or the robot has not a numeric positioning system, the relativemotion speed difference between the bridge and/or the trolley and thegripper as well as the swing angle of the gripper can be determined inrelation to a fixed laser light or laser lights 61, 62 reflected duringthe picture shots on the same image area 100 from the trolley above(see. FIGS. 5A and 5B). The above mentioned measures X′, Y′, X″ and Y″are thus calculated from this laser light reflection. The measuringresult obtained is directly the relative motion speed of the gripper inrelation to the corresponding motion directions of the crane. Becausethe load swings when the trolley or the other load carrying structuremoves, the swing speed of the load can be equal to the speed of thetrolley in both upper dead points (maximum swing angle α). When the loadis steadily in its place the above described positions can be separatedfrom each other e.g. by a laser light 61, 62 reflected down from thetrolley 4 observed by the identifiers 1 and 2, and the data obtained isconverted into the load swing angle data. The swing angle of the loadcan be accurately determined in relation to the upper load-bearingstructures even at small angle amplitudes. The laser light source isplaced in the trolley 4 as shown in FIGS. 5A and 5B. The swing angle αis in the upper picture unequal to zero and in the lower picture 5 it iszero. In both pictures in FIG. 5B the driving speed of the trolley isV_(trolley) in relation to the ground. The speed of the gripper swingmotion is V_(gripper), which in the upper picture varies between Vα andzero. The relative speed of the gripper is in FIG. 5 presented byV_(ground).

The digital image area data search can in some cases be facilitated bydefining fixed identifiable signs or lines for the bottom area.

In the absence of a numeric positioning system in the crane, the camerasaccording to the invention offer a better positioning also at manualoperation than e.g. is obtained with i.a. harmonic damping of the swingsystems based on the mathematical pendulum formula (T₀=2{square rootover ((1+L /g)}), in which T₀=swing time (s), 1=hoist cable length (m) ,g=9.807 m/s²). In manually operated cranes the swing damping methodsbased on mathematical formulas and default values are generallycharacterized by their over or under positioning when driven manually toa predetermined point. The invention lacks this feature, as when thedriver gives a stop command the swing is already in relation to theactual target area and the stop point is in the small image area 101filmed by the cameras 1 and 2. The cameras 1 and 2 and thus also thegripper 21, 22 can be positioned by teaching the control system or bythe data obtained during empirical research. The crane driver can bymeans of the machine vision system teach a crane lacking a numericpositioning the normal crane stop speeds and the requested decelerationdistances. When the deceleration distance has been learned, the machinevision system provides for a repetition of the deceleration taught bythe operator from the stop command to the point at a certain distance,and simultaneously performs the absorption of the load swing also withchanging load masses.

The motion speeds and the directions can be specified both in the bridgeand trolley direction. The slewing angle of the turning gripper can alsobe determined in relation to a chosen object.

The load absorption device of a container crane can also identify theset of numbers in the container end in order to secure the containercontent and loading address in the ship's hold. The automation increasesthe safety as the position of the containers in the ship arepre-determined for the stability of the ship. The same loadinginformation can be utilized in the unloading harbor with a similarequipment directly into the computer system of the receiving harbor,which makes the material handling more effective both in the dispatchingand the receiving harbor. Faults between the crane's or the robot's owninternal coordinates and other coordinates can be corrected. Theprocessor compares with the program the picture data received via thecamera with previously recorded values.

The gripper is directed by the camera to a previously known product bymeans of teaching, parametrization, characteristic features of thetarget or based on data provided by the CAD-image. An active positioningof a moving machine part is created in relation to a known or expectedtarget area and the positioning data of the moving machine part arecoded with this result, simultaneously compensating bending and twistingin the load carrying structures caused by different loading situations.

When teaching the gripper, areas of the camera view angle can beindicated from which the searched features of the product are to befound and save these in the memory, as shown in FIG. 6. The camera imagearea consists of parts. As can be seen from FIG. 6, the image areainspected can be restricted to a certain area and in this case to thearea restricted by h1′1, h1′2 and h2′1, h2′2. In FIG. 6 the containerE3,6 is placed in relation to the containers E3,1 and E3,2 thusutilizing the data about E3,2 in this area when placing the container.

When products are transferred based on the typesetting figure in thememory and the possible change made in the image of the previouslyfilmed area, the area in which to find the searched target isapproximately known. As this is the case, only part of the camerapicture can be reviewed. Each product has its own defined location point(deviation) in relation to the reference point. Based on this the craneis driven to the desired location according to predetermined locationpoints. The cameras are directed to review the selected area defined inthe deviation record of the product to be transferred and the knowntarget from a limited image area defined in the deviation record toinform the actual location site of the product in relation to thelocation sites of known products or other predetermined target. When thecamera has found the predetermined target, e.g. the angle of apreviously transferred product, the crane control system positions thetransferred product in relation to the found target.

The camera produces digital information which the computer programapplication utilizes in the gripper positioning. The camera comprises acamera and an application specific optic. Outdoors the camera is placedin a case, in which the window square in front of the camera lens isrotating preventing optic disturbances in the camera and also protectingthe camera lens from weather impacts. The case can be provided with aheating device. The camera can be a black-and-white or a color camera.The resolution of the camera can vary starting from 128×128 pixels up to1280×1024 pixels. As the final identification of the object is made fromclose and the object to be identified is generally big, the amount ofpixels can also be smaller. In situations where smaller text or bar codeis to be identified, a bigger pixel amount is required. In such cases itmight be economical to implement the gripper positioning and the text orthe bar code identification by parallel systems.

The objects to be identified can be classified in a small amount ofclasses according to some property (less than 10 classes) e.g. accordingto color. The colors are e.g. sorted in 256 different levels, the colorpictures can be divided according to the main colors in 3×256 differentlevels.

After the classification the image is pre-processed into a morepreferable form by digital image processing. After the pre-processingthe objects and their parts are to be segmented from their background.There are two different methods of segmentation: area-based and edgeidentification. In the area-based method the image is divided accordingto colors into homogenous fields. In the edge identification, steepcolor change points are searched in the image, i.e. area edges. Thesafety of the crane can also be improved by the system during thetransfer motions prior to the actual charge or discharge area. When theline of sight distance of the camera optic is adjusted to a distancewhich is twice to the crane stopping distance added with the computeroverall response time, the grippers can be driven when there is no sharpline, curve, etc. in the field of view, but the image is blurred. Whenone for the data system sudden, unexpected sharp uniform junctionappears in the image, the gripper motion is stopped.

The filming is intended to provide besides characteristics of the areasalso their mutual relations. There is a very accurate mathematical modelfor the identification of a known object. By concentrating to find fromthe picture material junctions (steep color changing points) andcomparing it to the model, an exact information about the location ofthe object in relation to the camera is obtained as a function ofdistance.

The state data of the camera optic at the teaching moment, i.a. focallength, distance, light are recorded simultaneously. With the controldata in the crane or robot memory as well as inquiries about the craneor robot real time state data, the camera optic can be adjustedaccording to the location information in the camera memory so thatidentification of the object is facilitated and functions reliably.

FIG. 8 presents a crane gripper intended to convey steel reels with thegrippers 21, 22 provided in one end of the crane beams 23, 24. Thecameras 1, 2 are placed in the crane beams 23, 24 close to the grippers,which cameras are turnable with a cylinder/piston 40 solution. One endof the cylinder or piston driver is supported in the crane beams 23, 24,and the other end of the cylinder is supported in the camera 1, 2. Thedeflection angle of the camera can normally be selected between 0-90degrees, i.e. between the horizontal or vertical plane. In horizontalplane the cameras are directed against each other, and in vertical planestraight downwards. When the gripper crane beams are to grip the steelreel 25, they are moved against each other and inside the reel,whereafter the reel can be lifted.

FIG. 9 presents a closer picture of a gripper attached to the cranebeam, and of the camera and the camera turning gear. Although thepicture shows a piston/cylinder turning gear, also others can be used.In FIG. 9 the camera 1 is shown in first position as an unbroken lineand in second position as a broken line, as also the cylinder/pistonunit, correspondingly. The bottom area is inspected with the downwarddirected camera when the goods are to be lowered. When the goods aregripped, the cameras are turned towards each other. The turning gearcomprises a piston/cylinder unit, one end of which is attached turnablyto the crane beam and the other end turnably to the pivot plate 28, towhich the camera 1 is attached. The pivot plate 28 is placed to turn inrelation to the crane beam and the gripper.

FIG. 10 presents another form of embodiment of the inspection of twodifferent image areas with the same camera. The camera 1, 2 is fixed tothe crane beam in the vicinity of the gripper. In front of the camera 1,2 image area can be placed a reflecting surface 30, i.e. a prism or amirror, which in oblique position gives an image of the gripper, i.e. ofthe horizontal plane. When the mirror 30 is turned by the turning gear31 in vertical position, the camera 1, 2 films its bottom area directlywithout the mirror 30. The picture of two areas can thus be reviewedwith the same camera. The reflecting surface can be provided withadditional properties, such as heating, etc.

As the planned system comprises alternatively two cameras located inrelation to each other in known sites, the object can be approached bythe triangulation principle. The objective is to identify by each camerapoints corresponding to each other located at different sites. Althoughthe picture material is 2-dimensional, this two camera stereo visionsystem provides also the location of the object, as the size, width ordiameter of the object are already known.

The identifiers can be placed in the gripper 21, 22, having an openingin the middle, to see through the opening of the other gripper 21, 22.One gripper can preferably be substituted by a light fixture 27, e.g.transversely in relation to the opening and with fluorescent lamps atthe end of the opening, and when the light of the observed area visiblefrom the camera picture matches the pre-determined one, the load can begripped with the gripper according to the information obtained from thepicture.

FIG. 7 shows the placing of the containers in relation to each other.The mathematical addresses of the containers in FIG. 7 are E 3,1 E 3,2,etc. in relation to the D-point. In FIG. 7 the container has been movedto site E 3,1 adjacent to which is placed the next container in site E3,2. The cameras 1, 2 attached to the grippers can be mounted in threedifferent positions. In the first position—the unbroken line—the camerais outside the long side of the container (E 3,2). The position of thecamera can be changed to e.g. outside the short side (pile edge on thelong side) or to above the container gripper (container fetch with emptygripper). The latter positions of the cameras are presented by brokenlines. Stepless turning gears can e.g. be used if other intermediatepositions are required. The conventional cylinder gearing clearances inthe camera attachment have been eliminated by coil springs acting in theopposite direction. The cameras can be furnished with light fixture(s)to maintain the light conditions essentially constant when taking thedifferent pictures.

FIG. 11 presents an example of a crane and camera system. Each camera isattached pivotably to the crane gripper or its vicinity. The systemcomprises two cameras 1, 2 which are pivotable with the pivotingelements 40. The control and adjustment of the cameras and the turningelements are implemented according to the instructions of the localcomputer 5 through the crane logic controller. The image signal producedby the cameras is transmitted straight to the computer video cards. Thecomputer 5 has a central processing unit 20, to which data transmissionbus has been connected for the data transmission i.a. a communicationcard 6, a computer net card 7, a sound card 8, a video card 10, a memoryunit 11, a hard disk 12 and a display card 13, which can communicatewith each other. According to the data received from the centralprocessing unit, the crane and the cameras can be controlled from thecontrol unit 14, but on the other hand also based on data obtained fromthe camera. The computer can be provided i.a. with a CD-station, userinterfaces (keyboard, microphone and loudspeakers, display), mass memoryand modem. A program has been installed in the computer mass memory andthe computer is connected to the control system. The computer operatingsystem is a so called multi processing operating system having thus inuse a multi media equipment. The control system controls the gripper orthe crane in real time in a pre-programmed way (logic controllerprogram). The control system comprises the logic control, the controls(forward, backward, right, left, slow, fast, etc), a digital positioningsystem and motor drives. The logic controller also attends to the realtime control and adjustment operations of the camera optic turningelements.

The computer analyzes the video picture and tells the logic controllerthrough a fast data transmission bus the deviation from the target pointand in which direction. The possible operator can also be given soundmessages, to easily show the I/O-data (the logic controllerinput/output-data) and to warn about risks, such as obstacles in thecrane motion track, or error situations.

A fast data transmission bus is available between the computer and thelogic controller. The computer has access to all data in the logiccontroller memory. If a fast data transmission is required, thetransmission can be based on a short macro-protocol using e.g. currentloop modems whereby the connection is straight and as fast as possible.

The loudspeakers connected to the computer enable the submitting ofsound messages to the driver. The loudspeaker control is implementedthrough the computer multimedia card. It can either happen so that priorto the speech recording a call is made through the sound card or therecorded text is converted into speech by the program through the soundcard, when there is an obstacle or due to some other pre-determinedcontrol. The gripper can be controlled with the microphone, when thesound is transformed into a signal comprehensible to the computer.

When a manually driven crane approaches the target area, the actualtarget area is reached with the machine vision system. The cranedeceleration and acceleration should be of that size that the changingbulk (load) to be transferred does not essentially change the cranedecelerations. If the bulk of the load would be decisive, the cranemight just slide on the rail when stopped too quickly. During theacceleration the trolley reels would just roll around as the starting ofthe inertial mass requires overcuming its own inertia. The known andgenerally in design used acceleration and deceleration values for cranesare 0.1-0.7 m/s², preferably 0.3-0.5 m/s₂. The acceleration anddeceleration of the robots are bigger, typically 1-4 m/s².

When the crane driver approaches the target area, he slows down thespeed close to 0.6 m/s (3-6 m/min). The crane designer has calculatedthe deceleration within the limits of the above mentioned acceleration.The final acceleration is determined by the chosen driving gear, andtherefore each crane has an application specific deceleration andacceleration. When the driver is an experienced crane driver, he drivesat a low speed and takes his hand off the crane direction controllerjust at the right moment to reach the correct stop place. According tothe crane's own deceleration, the crane continues still forward from thestop speed given by the driver along a relatively linear decelerationcurve down to the stop, unless the driver makes new correction motions.

The crane or robot swing observation system and the use of the swing tocontrol the speed are illustrated in a flow scheme in FIGS. 12A through12D. In the damping of the swing is checked that the camera is directeddownwards, the magnification ratio and the focusing distance arecorrect, the brightness is right and if laser is used, that it iscorrect. If everything is in order, the picture taken by the camera istransmitted to the digitizing card memory, the time when taking thepicture is recorded. Two clear edges of a certain area are searched bythe edge search. When the edges are found, the part of the picture to beinspected is determined, the edge features are thus included and savedin the memory. The next picture, which shooting moment is known, istransferred into the digitizing card memory. The recorded data of theprevious picture is retrieved from the new picture from an area definedby the motion direction. When the viewed area is found, the momentaryspeed of the gripper in relation to ground is calculated from thepictures, which informs the speed of the crane. The momentary speed ofthe gripper is obtained, which is the speed of the crane minus the speedof the gripper. When the result is obtained, speed correctioninstructions are given to the trolley and the bridge speed controlsystems. Suitable additional pictures can be utilized in differentstages.

A product marked with a label, e.g. bar code, provides directlyinformation about the size and desired location site of the load, whichcan be considered as reference data.

It should be considered that the invention has above been presented onlywith reference to some examples. The invention is however not in any wayto be considered restricted only to these solutions, the gripper e.g. isany gripping member or corresponding, but the invention includes thesolutions a man skilled in the art can carry out within the scope of theenclosed claims.

What is claimed is:
 1. A method for conveying a load between twolocation sites, at one of which sites the load is gripped for transfer,and at the other of which sites the load is discharged, said methodcomprising the steps of: selecting a load to be gripped; attaching atleast one grip member to the load; observing a first reference pointadjacent the load, at least with identification means that is disposedon the grip member and operably connected to a control system to recordinformation on the first reference point received from theidentification means; transferring the load toward the other locationsite, the other location site being defined by a second reference pointadjacent the other location site that is identifiable by theidentification means using information stored in the control system onthe second reference point; providing current information by observationwith the identification means at specified intervals during conveyanceof the load for use by the control system; and comparing the storedinformation on the second reference point to the current informationprovided by the identification means during conveyance of the load toidentify the second reference point, and to control the conveying of theload to the other location site.
 2. The method according to claim 1characterized in that the observation is carried out using video imagesand that the location of the load is determined by the comparison ofvideo images of one of the reference points taken at selected intervals.3. A method according to claim 1 characterized in that the load can haveits position at every point before, during and after transfer noted andrecorded by the identification means.
 4. A method according to claim 1characterized in that the identification means comprises at least onevideo camera attached to the grip member and operably connected to thecontrol system to provide image data on a reference point to the controlsystem, the image data including prior and current image data containedin and used by the control system to control conveyance of the load, theimage data having pixels, said method further comprising the steps ofselecting defined pixels of the current image data in the controlsystem, converting the current image data to current numerical data,transmitting the current numerical data to a computer, comparing thecurrent numerical data to the numerical data obtained from prior imagedata, and determining transit distance data that is subsequently used bythe control system to control conveyance of the load.
 5. A methodaccording to claim 4 characterized in that the prior image data isobtained by a method comprising the steps of taking pictures of areference point with the video camera and converting the prior imagedata or pictures into prior numerical data, subsequently taking at leastone picture of the reference point after a certain time interval,converting the current image data into current numerical data, anddetermining the transit distance data from a comparison of the prior andcurrent numeric data to enable the control system to use the transitdistance data to control the speed and direction of the conveyance ofthe load.
 6. A method for conveying a load between two location sites,at one of which sites the load is gripped for transfer, and at the otherof which sites the load is discharged, said method comprising the stepsof: selecting a load to be gripped; attaching at least one grip memberto the load; observing a first reference point adjacent the load with atleast one video camera that is disposed on the grip member and operablyconnected to a load control system to record information on thereference point received from the identification means; transferring theload to the other location site, the other location site being definedby a second reference point adjacent the other location site asidentified by the at least one video camera by taking a first picture ofthe other location site with the camera; storing the picture into acomputer memory and control system operably connected to the camera;determining in the first picture a digital image area to be searched;taking a second picture of the other location site with the camera; andcomparing the digital image area of the second picture with the digitalimage area of the first picture to determine the relative transitdistance from the position where the first picture was taken.
 7. Amethod according to claim 6 characterized in that the position of thecrane, the distance of the grip member from an object, and the speed ofthe trolley and/or the bridge are determined, and the speed of thetrolley and the bridge is adjusted according to relative transitdistance data.
 8. A method according to claim 7 characterized in thatthe distance of the grip member from the object is determined by takinga picture of the object.
 9. A method according to claim 4 characterizedin that the object, from which the distance of the grip member isdetermined, comprises one of the load, a feature of the load, or a laserbeam attached to the load.
 10. A method according to claim 7 furthercharacterized as determining the speed of the trolley and measuring theswing angle of the trolley or bridge using the camera and; adjusting thespeed of the trolley and/or bridge.
 11. A method according to claim 7further characterized as identifying the load while attaching the gripmember to the load.
 12. A method according to claim 6 characterized inthat, to determine the distance of an object from the camera, the camerais directed to a reference point placed at a known angle from the cameraand detectable in the image taken by the camera.
 13. A method accordingto claim 12 characterized in that the reference point is a laser beamapplied to the load.
 14. A method according to claim 12 characterized inthat the reference point is a distinguishing shape on the load.
 15. Amethod according to claim 7 characterized in that the camera ispivotally mounted to the grip member.
 16. A method according to claim 15characterized in that a mirror is pivotally mounted to the grip memberadjacent the camera to selectively enable the camera to take pictures ofobjects reflected towards the camera by the mirror.